Ascertaining the validity of individual protocols from Web-based personality inventories

The research described in this article estimated the relative incidence of protocols invalidated by linguistic incompetence, inattentiveness, and intentional misrepresentation in Webbased versus paper-and-pencil personality measures. Estimates of protocol invalidity were derived from a sample of 23,994 protocols produced by individuals who completed an online version of the 300-item IPIP representation of the NEO-PI-R (Goldberg, 1999). Approximately 3.8% of the protocols were judged to be products of repeat participants, many of whom apparently resubmitted after changing some of their answers. Among nonduplicate protocols, about 3.5% came from individuals who apparently selected a response option repeatedly without reading the item, compared to .9% in a sample of paper-and-pencil protocols. The missing response rate was 1.2%, which is 2–10 times higher than the rate found in several samples of paper-and-pencil inventories of comparable length. Two measures of response consistency indicated that perhaps 1% of the protocols were invalid due to linguistic incompetence or inattentive responding, but that Web participants were as consistent as individuals responding to a paper-and-pencil inventory. Inconsistency did not aVect factorial structure and was found to be related positively to neuroticism and negatively to openness to experience. Intentional misrepresentation was not studied directly, but arguments for a low incidence of misrepresentation are presented. Methods for preventing, detecting, and handling invalid response patterns are discussed. Suggested for future research are studies that assess the moderating eVects of linguistic incompetence, inattenPrepared for the special issue of the Journal of Research in Personality 39 (1), February 2005, containing the proceedings of the 2004 meeting of the Association for Research in Personality. E-mail address: [email protected]. 0092-6566/$ see front matter  2004 Elsevier Inc. All rights reserved. doi:10.1016/j.jrp.2004.09.009 104 J.A. Johnson / Journal of Research in Personality 39 (2005) 103–129 tiveness, and intentional misrepresentation on agreement between self-report and acquaintance judgments about personality.  2004 Elsevier Inc. All rights reserved.